Skip to content

Improve benchmark installation speed#2683

Closed
benjaminglass1 wants to merge 1 commit into
pytorch:mainfrom
benjaminglass1:improve_install_speed
Closed

Improve benchmark installation speed#2683
benjaminglass1 wants to merge 1 commit into
pytorch:mainfrom
benjaminglass1:improve_install_speed

Conversation

@benjaminglass1
Copy link
Copy Markdown
Contributor

@benjaminglass1 benjaminglass1 commented Apr 15, 2026 •
edited
Loading

This solves a problem noticed while installing benchmarks and watching htop. A significant amount of time per installation is wasted spinning up new Python subprocesses to repeatedly import packages and check their versions. This wastes substantial time when the package import takes several second (as with torch). This PR cuts down benchmark install time on my machine from 35 minutes to 12.

I also fixed one or two minor Ruff lints in the files I touched.

@meta-cla meta-cla Bot added the cla signed label Apr 15, 2026
@xmfan xmfan requested a review from a team April 15, 2026 23:14
@benjaminglass1
Copy link
Copy Markdown
Contributor Author

benjaminglass1 commented Apr 16, 2026

Test failures look the same as those on main.

@benjaminglass1
Improve benchmark installation speed
b97ab5a 
This solves a problem noticed while installing benchmarks and watching
htop.  A significant amount of time per installation is wasted checking
package version that have already been checked.  This commit caches
those versions.
@benjaminglass1 benjaminglass1 force-pushed the improve_install_speed branch from 22ab4a1 to b97ab5a Compare April 16, 2026 18:24
@benjaminglass1 benjaminglass1 requested a review from xmfan April 16, 2026 18:24
@benjaminglass1
Copy link
Copy Markdown
Contributor Author

benjaminglass1 commented Apr 16, 2026

@xmfan Please take a second look, I had to rewrite this after realizing this function is also used to validate that package versions haven't changed after installing benchmarks (which caching would invalidate). Currently getting some timing runs, but I still expect roughly the same speedup.

@benjaminglass1
Copy link
Copy Markdown
Contributor Author

benjaminglass1 commented Apr 16, 2026 •
edited
Loading

On my machine, installing all benchmarks on main takes roughly 35 minutes. With this change, installing all benchmarks takes roughly 12 minutes. These timings are obviously super variable, but that hopefully gives a sense of how impactful this change is.

The main reason for this is that we checked the version of several dependencies twice per benchmark. We got the version by importing the package in a new subprocess and querying __version__. For torch, this takes 2-3 seconds (on my machine), so that's a minute wasted every 10-15 packages. The new version queries PEP 566 metadata from the package in less than a millisecond, effectively eliminating the delay.

@meta-codesync
Copy link
Copy Markdown

meta-codesync Bot commented Apr 22, 2026

@huydhn has imported this pull request. If you are a Meta employee, you can view this in D102056803.

@huydhn huydhn requested a review from xuzhao9 April 22, 2026 22:22
xuzhao9
xuzhao9 reviewed Apr 22, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@xuzhao9 xuzhao9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@meta-codesync meta-codesync Bot closed this in 240ae9a Apr 23, 2026
@meta-codesync
Copy link
Copy Markdown

meta-codesync Bot commented Apr 23, 2026

@huydhn merged this pull request in 240ae9a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xuzhao9 xuzhao9 xuzhao9 left review comments

@huydhn huydhn huydhn approved these changes

@xmfan xmfan Awaiting requested review from xmfan

Assignees

No one assigned

Labels

cla signed Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@benjaminglass1 @huydhn @xuzhao9 @xmfan